Multiparty object recognition

ABSTRACT

Multiparty object recognition systems and methods are disclosed. A method of interactively manipulating virtual object data, wherein an object database is configured to store first party object data that corresponds to a first real-world object and is further configured to store second party object data that corresponds to a second real-world object, includes obtaining the first party object data and the second party object data for storage within the object database. Access to the object database is controlled such that the first party object data and the second party object data is accessible to the first party and the second party. Modification of the first party object data by the second party is facilitated to generate modified first party object data that is in accordance with at least one context parameter of the second party object data, and the modified first party object data is communicated to the first party.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/024,825, filed Jul. 15, 2014. The entire content of that applicationis hereby incorporated herein by reference.

TECHNICAL FIELD

The invention relates to virtual representation technologies, inparticular as they relate to modification of virtual objects and objectrepresentation attributes.

BACKGROUND

The background description includes information that may be useful inunderstanding the present invention. It is not an admission that any ofthe information provided herein is prior art or relevant to thepresently claimed invention, or that any publication specifically orimplicitly referenced is prior art.

Recent improvements in processing power have led to the possibility ofmainstream image recognition and object recognition applications. Forexample, gaming systems have been developed that incorporate elements ofimage recognition and object recognition to implement novel orsimplified control systems, where, for example, a user's movements arecaptured, digitized, and used to control movements or other actionswithin a game. Despite considerable potential, however, such imagerecognition and object recognition technology has not seen widespreadadoption.

SUMMARY

Various attempts have been made to incorporate elements of imagerecognition and object recognition systems in applications that allowfor the manipulation of digital images (or other digital analogs) ofreal-world objects. One such attempt is disclosed in United StatesPatent Application No. 2012/0306924 to Willoughby et al., whichdescribes the use of “skeletons” that are mapped onto both a user and anobject to allow a user to control movements of a representation of theobject via digitized images of the user. Another approach is describedin U.S. Pat. No. 7,564,469 to Cohen, which discloses the use of an imagecaptured by a mobile device to control a mixed reality display via acommand set. In another approach, United States Patent Application No.2012/0256954 to Soon-Shiong, describes an augmented reality system thatmodifies a scene that is presented to a user based on interactionsbetween the represented objects.

While such approaches have utility within certain narrow contexts, theyare too limited to be useful in many collaborative engineering andcreative applications, where efforts to portray augmented reality, mixedreality or virtual reality interactions between real-world objects arefrequently limited to previously prepared animations, the manipulationof physical models, and static whiteboard depictions. As such, there isstill a need for systems, apparatuses and methods for interactivelymanipulating virtual object data in augmented reality, mixed reality orvirtual reality environments, such as in real-time.

The inventive subject matter provides apparatuses, systems and methodsin which data related to real-world objects can be manipulated in acooperative manner by a plurality of users. Data related to a firstreal-world object and a second real-world object can be stored in adatabase that is accessible by two or more users via a portal engine.The portal engine permits a user to access and modify data related to afirst object, and further supports presentation of the modified firstobject data and data related to a second object to a different user. Thedata related to the first object can be modified in the context of datarelated to the second object.

In an embodiment, interactively manipulating virtual object data,wherein an object database is configured to store first party objectdata, associated with a first party, that corresponds to a firstreal-world object and is further configured to store second party objectdata, associated with a second party, that corresponds to a secondreal-world object, includes obtaining the first party object data andthe second party object data for storage within the object database. Thefirst party object data and the second party object data may include atleast one of digital image data, digital video data, digital audio data,digital text data, known virtual object data, biometric data, financialdata, medical data, and transaction data. Access to the object databaseis controlled such that the first party object data and the second partyobject data is accessible to at least the first party and the secondparty. Modification of the first party object data by the second partyis facilitated to generate modified first party object data, themodified first party object data being in accordance with at least onecontext parameter of the second party object data, and the modifiedfirst party object data is communicated to the first party. The at leastone context parameter may correspond to at least one of a temporalcontext, a spatial context, a thematic context, an additive context, asubtractive context, and a geometric context of the second party objectdata. The modified first party object data may be usable forpresentation within a collaborative user environment, and the modifiedfirst party object data may be communicated to the first party inreal-time. The first party and the second party may include at least oneof a person, a group of people, and an automated device.

In some embodiments, the modified first party object data may becommunicated as one of machine-readable image data or raw data which isconvertible at a receiving device.

In some embodiments, at least one of the first party object data andsecond party object data is analyzed, and analyzing at least one of thefirst party object data and second party object data may includematching features of the first party object data and the second partyobject data using at least one of an image recognition algorithm, apattern recognition algorithm, a speech recognition algorithm, a usermatching algorithm, and a location recognition algorithm.

In some embodiments, at least one of the first party object data andsecond party object data may comprise metadata. The metadata maycomprise at least one of first party-specific data or secondparty-specific data, location data, time data, user identification data,annotation data, social media data, product data, security data, andcontextual data.

In some embodiments, at least one of the first party object data and thesecond party object data may be obtained by using an image recognitionalgorithm on a digital representation of at least one of the firstreal-world object and the second real-world object.

In some embodiments, at least one of the first party object data and thesecond party object data may be obtained based on at least one of anidentification of object data by a user, and an instruction from anotherdevice.

In some embodiments, the first real-world object may include at leastone of a visible object, an audible object, a biometric object, amotion, and a gesture. The visible object may comprise at least one of aliving object, a landmark, an architectural structure, a conveyance, aconsumer product, a tool, a printed or displayed object, a toy, amusical instrument, an educational item, an art object, or a naturalinanimate object. The audible object may comprise at least one of avoice, a sound of a musical instrument, a sound of a living object, asound of an inanimate object, a sound of a vehicle, and anelectronically generated audible signal.

In some embodiments, the first real-world object and second real-worldobject may comprise different aspects of a single real-world object.

In some embodiments, the first real-world object may comprise at leastone property that triggers the modification of the first party objectdata, the at least one property comprising at least one of a chemicalsignature, temperature, pressure, electromagnetic field, radiofrequency,shape, color, texture, taste, smell, mechanical, biometric, orelectrical property.

In some embodiments, the modification of the first party object data maycomprise one or more of adding supplemental information, changing aspatial aspect of the first party object data, changing an attribute ofthe first party object data, combining the first party object data withthe second party object data, creating or changing a relationshipbetween the first party object data and the second party object data,and adding contextual data to the first party object data. In someembodiments, the modification of the first party object data may includeat least one of filtering, moving, manipulating, combining, editing,annotating, or animating the first party object data.

In some embodiments, the modified first party object data may compriseat least one of the second party object data, and object data from asource other than the first party and the second party.

In some embodiments, the object database may be accessible to at leastthe first party and the second party via a user portal, and the userportal may comprise at least one of a mobile device, an appliance, avehicle, a robot, a kiosk, a television, and a game console. A userinterface, comprising at least one of a graphical user interface, avoice-controlled interface, and a motion-controlled interface, may bepresented for access to the object database.

In some embodiments, access to the object database may be controlledsuch that the object database is accessible to a party other than thefirst party and second party. In some embodiments, access to the objectdatabase may be controlled based on at least one of a classification ofa real-world object, a proximity metric, a subscription status, and anauthorization.

In some embodiments, the modified first party object data may be storedwithin the object database, wherein the modified first object data maybe further modifiable to generate nth-order modified first object data.

In some embodiments, the modified first party object data may becommunicated to the second party.

Various objects, features, aspects and advantages of the inventivesubject matter will become more apparent from the following detaileddescription of preferred embodiments, along with the accompanyingdrawing figures in which like numerals represent like components.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary network environment that may be used formultiparty object recognition systems and methods in accordance withvarious embodiments.

FIG. 2 illustrates apparatuses that may be used for multiparty objectrecognition in accordance with an embodiment.

FIG. 3 illustrates a flow diagram of example operations for multipartyobject recognition in accordance with an embodiment.

FIG. 4A illustrates obtaining first party object data in accordance withan embodiment.

FIG. 4B illustrates obtaining second party object data in accordancewith an embodiment.

FIG. 5 illustrates modification of the first party object data by thesecond party in accordance with an embodiment.

FIG. 6 illustrates communicating the modified first party object data tothe first party in accordance with an embodiment.

FIG. 7 illustrates a flow diagram of example operations for multipartyobject recognition in accordance with an embodiment.

FIG. 8 illustrates a block diagram of a distributed computer system thatcan be used for implementing one or more aspects of the variousembodiments.

While the invention is described with reference to the above drawings,the drawings are intended to be illustrative, and other embodiments areconsistent with the spirit, and within the scope, of the invention.

DETAILED DESCRIPTION

The various embodiments now will be described more fully hereinafterwith reference to the accompanying drawings, which form a part hereof,and which show, by way of illustration, specific examples of practicingthe embodiments. This description may, however, be embodied in manydifferent forms and should not be construed as being limited to theembodiments set forth herein; rather, these embodiments are provided sothat this description will be thorough and complete, and will fullyconvey the scope of the invention to those skilled in the art. Amongother things, this description may be embodied as methods or devices.Accordingly, any of the various embodiments herein may take the form ofan entirely hardware embodiment, an entirely software embodiment or anembodiment combining software and hardware aspects. The followingdescription is, therefore, not to be taken in a limiting sense.

Throughout the description and claims, the following terms take themeanings explicitly associated herein, unless the context clearlydictates otherwise:

The phrase “in one embodiment” as used herein does not necessarily referto the same embodiment, though it may. Thus, as described below, variousembodiments of the invention may be readily combined, without departingfrom the scope or spirit of the invention.

As used herein, the term “or” is an inclusive “or” operator, and isequivalent to the term “and/or,” unless the context clearly dictatesotherwise.

The term “based on” is not exclusive and allows for being based onadditional factors not described, unless the context clearly dictatesotherwise.

As used herein, and unless the context dictates otherwise, the term“coupled to” is intended to include both direct coupling (in which twoelements that are coupled to each other contact each other) and indirectcoupling (in which at least one additional element is located betweenthe two elements). Therefore, the terms “coupled to” and “coupled with”are used synonymously. Within the context of a networked environmentwhere two or more components or devices are able to exchange data, theterms “coupled to” and “coupled with” are also used to mean“communicatively coupled with”, possibly via one or more intermediarydevices.

In addition, throughout the description, the meaning of “a,” “an,” and“the” includes plural references, and the meaning of “in” includes “in”and “on.”

The following discussion provides many example embodiments of theinventive subject matter. Although some of the various embodimentspresented herein constitute a single combination of inventive elements,it should be appreciated that the inventive subject matter is consideredto include all possible combinations of the disclosed elements. As such,if one embodiment comprises elements A, B, and C, and another embodimentcomprises elements B and D, then the inventive subject matter is alsoconsidered to include other remaining combinations of A, B, C, or D,even if not explicitly discussed herein.

As used in the description herein and throughout the claims that follow,when a system, engine, server, device, module, or other computingelement is described as being configured to perform or execute functionson data in a memory, the meaning of “configured to” or “programmed to”is defined as one or more processors or cores of the computing elementbeing programmed by a set of software instructions stored in the memoryof the computing element to execute the set of functions on target dataor data objects stored in the memory.

It should be noted that any language directed to a computer should beread to include any suitable combination of computing devices, includingservers, interfaces, systems, databases, agents, peers, engines,controllers, modules, or other types of computing device structuresoperating individually or collectively. One skilled in the art shouldappreciate the computing devices comprise a processor configured toexecute software instructions stored on a tangible, non-transitorycomputer readable storage medium (e.g., hard drive, FPGA, PLA, solidstate drive, RAM, flash, ROM, etc.). The software instructions configureor program the computing device to provide the roles, responsibilities,or other functionality as discussed below with respect to the disclosedapparatus. Further, the disclosed technologies can be embodied as acomputer program product that includes a non-transitory computerreadable medium storing the software instructions that causes aprocessor to execute the disclosed steps associated with implementationsof computer-based algorithms, processes, methods, or other instructions.In some embodiments, the various servers, systems, databases, orinterfaces exchange data using standardized protocols or algorithms,possibly based on HTTP, HTTPS, AES, public-private key exchanges, webservice APIs, known financial transaction protocols, or other electronicinformation exchanging methods. Data exchanges among devices can beconducted over a packet-switched network, such as the Internet, a LAN,WAN, VPN, or other type of packet-switched network, a circuit-switchednetwork, cell-switched network; or other type of network.

The following description includes information that may be useful inunderstanding the present invention. It is not an admission that any ofthe information provided herein is prior art or relevant to thepresently claimed invention, or that any publication specifically orimplicitly referenced is prior art.

In some embodiments, the numbers expressing quantities of ingredients,properties such as concentration, reaction conditions, and so forth,used to describe and claim certain embodiments of the invention are tobe understood as being modified in some instances by the term “about.”Accordingly, in some embodiments, the numerical parameters set forth inthe description and attached claims are approximations that can varydepending upon the desired properties sought to be obtained by aparticular embodiment. In some embodiments, the numerical parametersshould be construed in light of the number of reported significantdigits and by applying ordinary rounding techniques. Notwithstandingthat the numerical ranges and parameters setting forth the broad scopeof some embodiments of the invention are approximations, the numericalvalues set forth in the specific examples are reported as precisely aspracticable. The numerical values presented in some embodiments of theinvention may contain certain errors necessarily resulting from thestandard deviation found in their respective testing measurements.

Unless the context dictates the contrary, all ranges set forth hereinshould be interpreted as being inclusive of their endpoints andopen-ended ranges should be interpreted to include only commerciallypractical values. Similarly, all lists of values should be considered asinclusive of intermediate values unless the context indicates thecontrary.

As used in the description herein and throughout the claims that follow,the meaning of “a,” “an,” and “the” includes plural reference unless thecontext clearly dictates otherwise. Also, as used in the descriptionherein, the meaning of “in” includes “in” and “on” unless the contextclearly dictates otherwise.

The recitation of ranges of values herein is merely intended to serve asa shorthand method of referring individually to each separate valuefalling within the range. Unless otherwise indicated herein, eachindividual value is incorporated into the description as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g. “such as”) provided with respectto certain embodiments herein is intended merely to better illuminatethe invention and does not pose a limitation on the scope of theinvention otherwise claimed. No language in the description should beconstrued as indicating any non-claimed element essential to thepractice of the invention.

Groupings of alternative elements or embodiments of the inventiondisclosed herein are not to be construed as limitations. Each groupmember can be referred to and claimed individually or in any combinationwith other members of the group or other elements found herein. One ormore members of a group can be included in, or deleted from, a group forreasons of convenience and/or patentability. When any such inclusion ordeletion occurs, the description herein is deemed to contain the groupas modified, thus fulfilling the written description of all Markushgroups used in the appended claims.

One should appreciate that the disclosed techniques provide manyadvantageous technical effects including facilitating and streamliningcooperative design and editing efforts. Thus, a computing device isconfigured to allow data related to real-world objects to be manipulatedin a cooperative manner by two or more users. The computing device isable to process substantial amounts of digital data well beyond thecapability of a human being.

A multiparty object recognition as described herein allows for themanipulation of virtual objects by multiple users. Methods and systemsfor a multiparty object recognition can be implemented in a variety ofways. FIG. 1 illustrates an exemplary network environment that may beused for multiparty object recognition systems and methods in accordancewith various embodiments. In one exemplary embodiment, a multipartyobject recognition system for interactively manipulating object data canbe implemented within a networked computing environment 100 (e.g., acloud-computing environment). Networked computing environment 100comprises at least one network 102. In an exemplary embodiment, network102 is the Internet. In other embodiments, network 102 may comprise oneor more of a number of different types of networks, such as, forexample, an intranet, a local area network (LAN), a wide area network(WAN), a wireless network, or other type of packet-switched network,circuit-switched network or cell-switched network. Other networks may beused. Alternatively, network 102 may comprise a combination of differenttypes of networks.

Within the environment 100, one or more servers 106A-B and/or clientdevices 104A-E can implement a multiparty object recognition system.Examples of client devices 104A-E can include mobile devices (such aslaptop computers, tablets, or smartphones), appliances, vehicles,robots, kiosks or other types of public terminals, televisions, digitalmedia players, game consoles, or other types of computing devices.Further, a client device may be associated with a party (user) suchthat, for example, client devices 104A and 104B may be associated with afirst party and a second party, respectively.

In an embodiment, one or more client devices 104A-E and/or servers106A-B can host or be in communication with apparatuses, e.g., portalengine 110 and object database 120, to implement a multiparty objectrecognition system that is accessible to client devices 104A-E. Forexample, object database 120 may be configured to store first partyobject data, associated with a first party, that corresponds to a firstreal-world object and be further configured to store second party objectdata, associated with a second party, that corresponds to a secondreal-world object. Portal engine 110 may be configured to obtain thefirst party object data and the second party object data from any ofclient devices 104A-E and servers 106A-B for storage within objectdatabase 120. For example, portal engine 110 may be configured to obtainobject data, including object data associated with a first party (i.e.,first party object data) and a second party (i.e., second party objectdata), that corresponds to real-world objects depicted in a digitalrepresentation (e.g., a digital image or a video stream) using one ormore image recognition algorithms, or based on identificationinstructions provided by a user or an external device.

Portal engine 110 may be further configured to control access to objectdatabase 120 such that the first party object data and the second partyobject data is accessible to the first party (e.g., at client device104A) and the second party (e.g., at client device 104B), facilitatemodification of the first party object data by the second party togenerate modified first party object data that is in accordance with atleast one context parameter of the second party object data, andcommunicate the modified first party object data to the first party,e.g., at client device 104A or another client device accessible by thefirst party.

As shown, object database 120 is communicatively coupled to portalengine 110. In an embodiment, portal engine 110 is configured togenerate a user portal to object database 120 that may be accessible tomultiple parties via client devices 104A-E. The user portal can beprogrammed or configured by the portal engine 110 to provide access todata stored in object database 120.

Alternatively, object database 120 and portal engine 110 can be remotefrom each other as well as from the one or more client devices 104A-Eused to access object database 120, e.g., via the user portal. Forexample, one or more of object database 120 and portal engine 110 can behoused in one or more server computers 106A-B located remotely fromclient devices 104A-E. Likewise, the server(s) (or other computingdevice(s)) hosting portal engine 110 can be located remotely from servercomputers or other non-transitory computer-readable storage mediahosting object database 120. In some embodiments, one or more of objectdatabase 120 and portal engine 110 can be integral to one or more ofclient devices 104A-E. For example, one of the client devices, e.g.,client device 104A, may host portal engine 110 and/or object database120, such that the other client devices 104B-E are clients to the hostclient device. In other embodiments, various functions described hereinas being performed by portal engine 110 and object database 120 may bedistributed between a plurality of client devices 104A-E, a plurality ofservers 106A-B, or any combination of client devices and servers.

FIG. 2 illustrates apparatuses that may be used for multiparty objectrecognition in accordance with an embodiment. In block diagram 200,elements for multiparty object recognition include portal engine 110,object database 120, and client devices 104A-B in communication withportal engine 110 via network 102. However, it should be noted that theelements in FIG. 2, and the various functions attributed to each of theelements, while exemplary, are described as such solely for the purposeof ease of understanding. One skilled in the art will appreciate thatone or more of the functions ascribed to the various elements may beperformed by any one of the other elements, and/or by an element (notshown) configured to perform a combination of the various functions.

In an embodiment, interactively manipulating virtual object data,wherein object database 120 is configured to store first party objectdata, associated with a first party, that corresponds to a firstreal-world object and is further configured to store second party objectdata, associated with a second party, that corresponds to a secondreal-world object, includes portal engine 110 being configured to obtainthe first party object data and the second party object data for storagewithin object database 120. The first party object data and the secondparty object data may include at least one of digital image data,digital video data, digital audio data, digital text data, known virtualobject data, biometric data, financial data, medical data, andtransaction data. In some embodiments, the first party object data andsecond party object data may comprise metadata. The metadata maycomprise at least one of first party-specific data orsecond-party-specific object data, location data, time data, useridentification data, annotation data, social media data, product data,security data, and contextual data.

In some embodiments, portal engine 110 may be configured to obtain atleast one of the first party object data and the second party objectdata by using an image recognition algorithm on a digital representationof at least one of the first real-world object and the second real-worldobject. In some embodiments, the feature detection algorithm may includeat least one of a scale-invariant feature transform (SIFT), Fast RetinaKeypoint (FREAK), Histograms of Oriented Gradient (HOG), Speeded UpRobust Features (SURF), DAISY, Binary Robust Invariant ScalableKeypoints (BRISK), FAST, Binary Robust Independent Elementary Features(BRIEF), Harris Corners, Edges, Gradient Location and OrientationHistogram (GLOH), Energy of image Gradient (EOG) or Transform InvariantLow-rank Textures (TILT) feature detection algorithm. Portal engine 110also may be configured to obtain at least one of the first party objectdata and the second party object data based on at least one of anidentification of object data by a user, and an instruction from anotherdevice.

In some embodiments, the first real-world object and second real-worldobject may include at least one of a visible object, an audible object,a biometric object, a motion, and a gesture. For example, the visibleobject may comprise at least one of a living object, a landmark, anarchitectural structure, a conveyance, a consumer product, a tool, aprinted or displayed object, a toy, a musical instrument, an educationalitem, an art object, or a natural inanimate object, and the audibleobject may comprise, e.g., at least one of a voice, a sound of a musicalinstrument, a sound of a living object, a sound of an inanimate object,a sound of a vehicle, and an electronically generated audible signal. Insome embodiments, the first real-world object and second real-worldobject may comprise different aspects of a single real-world object. Forexample, the first real-world object may be one side of a real-worldobject and the second real-world object may be another side of thereal-world object.

Portal engine 110 is further configured to control access to objectdatabase 120 such that the first party object data and the second partyobject data is accessible to at least the first party and the secondparty, e.g., via client devices 104A and 104B, respectively. The firstparty and the second party may include at least one of a person, a groupof people, and an automated device.

Portal engine 110 is further configured to facilitate modification ofthe first party object data by the second party (e.g., via client device104B) to generate modified first party object data, the modified firstparty object data being in accordance with at least one contextparameter of the second party object data. The at least one contextparameter may correspond to at least one of a temporal context, aspatial context, a thematic context, an additive context, a subtractivecontext, and a geometric context of the second party object data.

In some embodiments, the first real-world object may comprise at leastone property that triggers the modification of the first party objectdata, the at least one property comprising at least one of a chemicalsignature, temperature, pressure, electromagnetic field, radiofrequency,shape, color, texture, taste, smell, mechanical, biometric, orelectrical property.

In some embodiments, the modification of the first party object data maycomprise one or more of adding supplemental information, changing aspatial aspect of the first party object data, changing an attribute ofthe first party object data, combining the first party object data withthe second party object data, creating or changing a relationshipbetween the first party object data and the second party object data,and adding contextual data to the first party object data. In someembodiments, the modification of the first party object data may includeat least one of filtering, moving, manipulating, combining, editing,annotating, or animating the first party object data.

In some embodiments, the modified first party object data may compriseat least one of the second party object data, and object data from asource other than the first party and the second party.

Portal engine 110 is further configured to communicate the modifiedfirst party object data to the first party. The modified first partyobject data may be usable for presentation within a collaborative userenvironment, and the modified first party object data may becommunicated to the first party in real-time. In some embodiments, themodified first party object data may be communicated as one ofmachine-readable image data or raw data which is convertible at areceiving device. In some embodiments, the modified first party objectdata also may be communicated to the second party.

In some embodiments, portal engine 110 is further configured to analyzeat least one of the first party object data and second party objectdata. For example, portal engine 110 may be configured to analyze atleast one of the first party object data and second party object data bymatching features of the first party object data and the second partyobject data using at least one of an image recognition algorithm, apattern recognition algorithm, a speech recognition algorithm, a usermatching algorithm, and a location recognition algorithm.

In some embodiments, portal engine 110 may be further configured toprovide at least the first party and the second party access to theobject database via a user portal. For example, the user portal maycomprise at least one of a mobile device (e.g., client devices 104A-B),an appliance, a vehicle, a robot, a kiosk, a television, and a gameconsole. In some embodiments, portal engine 110 may be configured topresent a user interface, comprising at least one of a graphical userinterface, a voice-controlled interface, and a motion-controlledinterface, to the first party and the second party for access to theobject database. For example, the user portal and/or user interface maycomprise a content source control system (e.g., such as GitHub, theConcurrent Versions System (CVS), etc.), so that the content from thefirst party and the second party can be synchronized once it ispublished, e.g., by portal engine 110.

In some embodiments, portal engine 110 may be further configured tocontrol access to object database 120 such that the object database isaccessible to a party other than the first party and second party. Insome embodiments, access to object database 120 may be controlled basedon at least one of a classification of a real-world object, a proximitymetric, a subscription status, and an authorization.

In some embodiments, portal engine 110 may be further configured tostore the modified first party object data within object database 120,wherein the modified first object data may be further modifiable togenerate nth-order modified first object data.

FIG. 3 illustrates a flow diagram of example operations for multipartyobject recognition in accordance with an embodiment. FIG. 3 presents anoverview of a multiparty object recognition method 300 executed by aportal engine (e.g., computer server, computer client, tablet, gamingconsole, etc.), such as portal engine 110 in FIG. 2. FIGS. 4-6 belowpresent additional details regarding the various steps of method 300,and are referenced seriatim with corresponding steps of method 300.

At step 302, interactively manipulating virtual object data, wherein anobject database is configured to store first party object data,associated with a first party, that corresponds to a first real-worldobject and is further configured to store second party object data,associated with a second party, that corresponds to a second real-worldobject, includes obtaining the first party object data and the secondparty object data for storage within the object database. FIG. 4Aillustrates obtaining first party object data in accordance with anembodiment. In block diagram 400A, an image of first real-world object402 is captured by image capture device 404 and presented as digitalrepresentation 406 (e.g., a digital image) within user interface 408.For example, a client device, such as client device 104A, may compriseimage capture device 404 (e.g., a digital camera or video camera) anduser interface 408 (e.g., a visual display).

First real-world object 402 may include at least one of a visibleobject, an audible object, a biometric object, a motion, and a gesture.For example, a visible real-world object may comprise at least one of aliving object, a landmark, an architectural structure, a conveyance, aconsumer product, a tool, a printed or displayed object, a toy, amusical instrument, an educational item, an art object, or a naturalinanimate object. An audible real-world object may comprise, e.g., atleast one of a voice, a sound of a musical instrument, a sound of aliving object, a sound of an inanimate object, a sound of a vehicle, andan electronically generated audible signal.

In an embodiment, portal engine 110 is configured to receive digitalrepresentation 406 and obtain first party object data by using an imagerecognition algorithm on digital representation 406. For example, theimage recognition algorithm may be a feature detection algorithmincluding at least one of a scale-invariant feature transform (SIFT),Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG),Speeded Up Robust Features (SURF), DAISY, Binary Robust InvariantScalable Keypoints (BRISK), FAST, Binary Robust Independent ElementaryFeatures (BRIEF), Harris Corners, Edges, Gradient Location andOrientation Histogram (GLOH), Energy of image Gradient (EOG) orTransform Invariant Low-rank Textures (TILT) feature detectionalgorithm. Alternatively, portal engine 110 may be configured to obtainthe first party object data based on at least one of an identificationof object data by a user (e.g., the first party), and an instructionfrom another device. In an embodiment, the first party object data mayinclude at least one of digital image data, digital video data, digitalaudio data, digital text data, known virtual object data, biometricdata, financial data, medical data, and transaction data. In someembodiments, the first party object data may comprise metadata. Themetadata may comprise at least one of first party-specific data,location data, time data, user identification data, annotation data,social media data, product data, security data, and contextual data.

Portal engine 110 is then configured to store the first party objectdata in object database 120.

FIG. 4B illustrates obtaining second party object data in accordancewith an embodiment. Similar to block diagram 400A above, in blockdiagram 400B an image of second real-world object 410 is captured byimage capture device 412 and presented as digital representation 414(e.g., a digital image) within user interface 416. For example, a clientdevice associated with a second user, such as client device 104B, maycomprise image capture device 412 (e.g., a digital camera or videocamera) and user interface 416 (e.g., a visual display).

Second real-world object 410 may include at least one of a visibleobject, an audible object, a biometric object, a motion, and a gesture.For example, a visible real-world object may comprise at least one of aliving object, a landmark, an architectural structure, a conveyance, aconsumer product, a tool, a printed or displayed object, a toy, amusical instrument, an educational item, an art object, or a naturalinanimate object. An audible real-world object may comprise, e.g., atleast one of a voice, a sound of a musical instrument, a sound of aliving object, a sound of an inanimate object, a sound of a vehicle, andan electronically generated audible signal.

In an embodiment, portal engine 110 is configured to receive digitalrepresentation 414 and obtain second party object data by using an imagerecognition algorithm on digital representation 414. For example, theimage recognition algorithm may be a feature detection algorithmincluding at least one of a scale-invariant feature transform (SIFT),Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient (HOG),Speeded Up Robust Features (SURF), DAISY, Binary Robust InvariantScalable Keypoints (BRISK), FAST, Binary Robust Independent ElementaryFeatures (BRIEF), Harris Corners, Edges, Gradient Location andOrientation Histogram (GLOH), Energy of image Gradient (EOG) orTransform Invariant Low-rank Textures (TILT) feature detectionalgorithm. Alternatively, portal engine 110 may be configured to obtainthe second party object data based on at least one of an identificationof object data by a user (e.g., the second party), and an instructionfrom another device. In an embodiment, the second party object data mayinclude at least one of digital image data, digital video data, digitalaudio data, digital text data, known virtual object data, biometricdata, financial data, medical data, and transaction data. In someembodiments, the second party object data may comprise metadata. Themetadata may comprise at least one of second party-specific data,location data, time data, user identification data, annotation data,social media data, product data, security data, and contextual data.

Portal engine 110 is then configured to store the second party objectdata in object database 120. In some embodiments, the first real-worldobject and second real-world object may comprise different aspects of asingle real-world object. For example, the first real-world object maybe one side of a real-world object and the second real-world object maybe another side of the real-world object.

At step 304, access to object database 120 is controlled such that thefirst party object data and the second party object data is accessibleto at least the first party and the second party, e.g., via clientdevices 104A and 104B, respectively. The first party and the secondparty may include at least one of a person, a group of people, and anautomated device.

At step 306, modification of the first party object data by the secondparty (e.g., via client device 104B) is facilitated to generate modifiedfirst party object data, the modified first party object data being inaccordance with at least one context parameter of the second partyobject data. FIG. 5 illustrates modification of the first party objectdata by the second party in accordance with an embodiment. In blockdiagram 500, once access is granted to object database 120 the secondparty may modify the first party object data (e.g., via user interface416) in accordance with a context parameter of the second party objectdata (e.g., modifications 504 and 506), thereby generating modifiedfirst party object data. The second party can modify the first partyobject data in any suitable fashion. For example, the modification ofthe first party object data may comprise one or more of addingsupplemental information, changing a spatial aspect of the first partyobject data, changing an attribute of the first party object data,combining the first party object data with the second party object data(e.g., as shown by modifications 405 and 506), creating or changing arelationship between the first party object data and the second partyobject data, and adding contextual data to the first party object data.In some embodiments, the modification of the first party object data mayinclude at least one of filtering, moving, manipulating, combining,editing, annotating, or animating the first party object data. Suchmodified first party object data can include second party object data.For example, data representing the first real-world object 402 suppliedby the first party can be modified in accordance with a contextparameter of data representing the second real-world object 410 suppliedby the second party such that, when displayed, a representation of thefirst real-world object 402 can appear to be manipulated or otherwisemodified by a representation of the second real-world object 410, asshown in representation 502. Alternatively, modified first party objectdata can include object data from a source other than the first partyand/or second party, for example object data from a third party. The atleast one context parameter may correspond to at least one of a temporalcontext, a spatial context, a thematic context, an additive context, asubtractive context, a geometric context of the second party objectdata, or a combination of context parameters.

In some embodiments, the first real-world object may comprise at leastone property that triggers the modification of the first party objectdata, the at least one property comprising at least one of a chemicalsignature, temperature, pressure, electromagnetic field, radiofrequency,shape, color, texture, taste, smell, mechanical, biometric, orelectrical property.

At step 308, the modified first party object data is communicated to thefirst party. FIG. 6 illustrates communicating the modified first partyobject data to the first party in accordance with an embodiment. Inblock diagram 600, portal engine 110 can be configured to retrieve themodified first party object data (e.g., from local memory and/or objectdatabase 120) and communicate the modified first party object data(e.g., via network 102 using an output module) for presentation 602 tothe first party, such as at user interface 408. In some embodiments, themodified first party object data also may be communicated to the secondparty, e.g., for presentation at client device 104B. As such, themodified first party object data may be usable for presentation within acollaborative user environment (e.g., for presentation at user interface408 and user interface 416), and the modified first party object datamay be communicated at least to the first party in real-time. In someembodiments, the modified first party object data may be communicated asone of machine-readable image data or raw data which is convertible at areceiving device, e.g., client device 104A. In various embodiments, thefirst party can also modify the second party object data in a similarfashion, thereby generating modified second party object data in thecontext of first party object data. For example, portal engine 110 canpermit the first user to access object database 120 and modify firstparty object data in the context of second party object data in order togenerate modified first party object data. Such modifications to thefirst party object data and/or the second party object data can be madeprior to (long-term) storage of the unmodified first party object dataand/or second party object data. It should be appreciated that in someembodiments modified first party object data and modified second partyobject data can be further modified repeatedly in an iterative process.

Therefore, portal engine 110 is configured to facilitate modification,by a user, of object data provided by a different user. For example,utilizing a user portal, a second party User B can modify object datathat was supplied by a first party User A (i.e. first party object data)to produce modified first party object data. Subsequently, portal engine110 can present this modified first party object data to User A and/orUser B, along with second party object data associated with User B. Insome embodiments, the portal engine and/or the object database can beused to store such modified first party object data. Modifications ofthe first party object data can include the addition of supplementalinformation, changing one or more spatial aspects of first party objectdata, changing an attribute of a first party object data, combiningfirst party object data with second party object data, creating arelationship between the first party object data and second party objectdata, changing a relationship between the first party object data andsecond party object data, and adding contextual data to the first partyobject data.

Towards this end, a real-world object from which first party objectdata, second party object data, or both first party object data andsecond party object data is derived can include one or more propertiesthat trigger or are amenable to modification. For example, suchproperties can include a chemical signature, a temperature, a pressure,an electromagnetic field, a polarity, a static charge, a radiofrequency,a shape or configuration, a color, a texture, a taste, a smell, amechanical property, a biometric property, and/or an electricalproperty. In some embodiments, a real-world object can include a numberof parts. For example, each part may exhibit one or more of suchmodifiable properties, and can be modified independently. In otherembodiments, modification of data related to one property can trigger anautomated modification of data related to a different property. Forexample, modification of data related to shape or configuration of areal-world object by a user can trigger an automated change in datarelated to a mechanical property that reflects a result of theuser-directed change.

Further, object database 120 is configured to store data related toreal-world objects and/or modified data related to real-world objects.Such a database can be supported by any computing device that supportsdata storage (such as a mainframe, server, desktop computer, and/orlaptop computer), and may be located remotely from other systemcomponents. In some embodiments, object database 120 can be composed oftwo or more secondary databases that are in communication with eachother. Such secondary databases can reside in multiple and/or separatecomputing devices. The object database 120 is in communication withportal engine 110, which can provide access to object database 120 fortwo or more users. In some embodiments that utilize two or moresecondary object databases, the secondary object databases can be incommunication with each other and/or with portal engine 110. Portalengine 110 may be located at a distance from object database 120, andcommunication between object database 120 and portal engine 110 can bethrough a wired connection, a wireless connection, or via anintermediary data system such as a data cloud or cloud storage. In someembodiments, portal engine 110 can be located in a portable device, suchas a laptop computer, tablet, and/or smart phone. At least two (and insome embodiments, three or more) parties, may be provided access toobject database 120 by portal engine 110.

The various embodiments allow interactive manipulation of a virtualobject using a common or shared platform. First party object data, whichcorresponds to a first real world object and is obtained from a firstuser or party, and second party object data, which corresponds to asecond real world object and is obtained from a second user or party, isstored in an object database. The first party and second party (and insome embodiments, additional users) are provided with access to objectdatabase 120 via portal engine 110. Such a portal engine can be locatedin or on a user device that is remote or distal to the object database.Examples of user devices include a desktop computer, a laptop computer,a tablet, and/or a smart phone. In some embodiments, access to objectdatabase 120 can be controlled by physical access to a terminal orsimilar controller, a password, biometric data, possession of a hardwareand/or software key, the class or other characteristic of the real worldobject, a proximity metric (ex: a location), a subscription status,and/or an authorization. To facilitate communication and usability, aportal engine can include a user interface. Suitable user interfacesinclude a graphical user interface, a voice-controlled interface, and/ora motion controlled interface.

In the various embodiments, modified first party object data can befurther modified to form secondary, tertiary, and nth-order modifiedfirst party object data. Such a series of data modifications can bedisplayed, for example, in an order that creates an illusion of movementor a passage of time.

In some embodiments, portal engine 110 can be configured to perform ananalysis on the first party object data, the second party object data,and/or both the first party object data and the second party objectdata. For example, portal engine 110 can match features of the firstparty object data and the second party object data. Alternatively, theportal engine can utilize time coding, image recognition, patternrecognition, speech recognition, user matching, user relationship, andor location or geographic recognition in the analysis of object data.

In some embodiments, some or all of the information provided by the userdevices to the portal engine and/or other user devices can be providedmore than once. For example, location data of the user devices can beprovided to the portal engine and/or each other multiple times (e.g.,continuously, repeatedly according to a desired frequency), and as suchthe object data contributed by/corresponding to each user device can beupdated to account for this changed location. Likewise, informationabout a particular scene as determined by each user device can be sharedfrequently between the user devices themselves as well as the portalengine such that the portal engine and the other devices can receiveupdated regarding changes in a scene.

As the object data in the object database is updated by a correspondinguser device, the updated version of the object data is shared with otheruser devices in the virtual space. For example, a first user device cantransmit video data about a physical structure, whereby the virtualversion of the structure can begin with only the initial view of thestructure as provided by the first user device, and whereby the virtualversion of the structure is updated as the user continues to transmitvideo data showing different parts of the structure. Other users, thencan use their own object data (e.g., annotations for or about astructure that is being worked on, their own virtual tools; etc.) on thevirtual structure as it becomes more developed. In some embodiments, anymodifications to object data by other users in the collaborativeenvironment can be updated, removed, deleted, modified, or otherwisechanged based on updated versions of the source object data.

In a further aspect, the updating of object data by the source device orother user devices can be used in a dynamic recognition system. Asobject data is updated over time with more information, the object databecomes a more “fleshed out” representation of the real-world object. Asthe object data becomes more “fleshed out”, the recognition of thereal-world object and its properties can be updated to recognize new ordifferent aspects of the real-world object, to further refine therecognized real-world object (such as from a general recognition of acar in an image as a generic ‘car’ to increasingly specific recognitionsof ‘sedan’ to ‘sedan of certain make and model’ to ‘sedan of certainmake and model of a particular year’).

In situations where multiple users are working in a collaborativeenvironment, it is possible that their contributions can be duplicitous,redundant, or overlapping. In such cases, portal engine 110 can beconfigured to resolve redundancies in provided information, such asredundancies in data objects or redundancies in interactions with ormodifications to data objects, such that a synchronized, synthesizedversion of the object data is shared among all participants.

As the object data representing real-world objects, scenes, etc., areupdated over time by one or more user devices and collaborations, thephysical aspects of the real-world objects, scenes, etc., (e.g.,rigidity, static/dynamic properties, opacity, specularity, etc.) can beretained in object database 120. Changes to the object data within theportal engine, such as by other users interacting with the object datain a collaborative environment, can also be tracked as they affect thephysical aspects represented by the virtual version of the real-worldobject. The portal engine can track the changes and resolve the physicalchanges over time.

FIG. 7 illustrates a flow diagram of example operations for multipartyobject recognition in accordance with an embodiment. FIG. 7 presents anoverview of a multiparty object recognition method 700 executed by aportal engine (e.g., computer server, computer client, tablet, gamingconsole, etc.), such as portal engine 110 in FIG. 2.

At step 702, multiparty object recognition includes receiving digitalrepresentations from a first party and a second party. For example,portal engine 110 may receive one or more digital representations of afirst real-world object (e.g., as captured by the first party using animage capture device, as illustrated in FIG. 4A) and one or more digitalrepresentations of a second real-world object (e.g., as captured by thesecond party using an image capture device, as illustrated in FIG. 4B).

At step 704, if object data is included with the digital representationsreceived from the first party and the second party, the first partyobject data and second party object data may be stored (e.g., in objectdatabase 120) at step 708. If object data is not included with thedigital representations received from either the first party or thesecond party, first party object data and/or second party object datamay be generated using an image recognition algorithm at step 706. Forexample, the image recognition algorithm may be a feature detectionalgorithm including at least one of a scale-invariant feature transform(SIFT), Fast Retina Keypoint (FREAK), Histograms of Oriented Gradient(HOG), Speeded Up Robust Features (SURF), DAISY, Binary Robust InvariantScalable Keypoints (BRISK), FAST, Binary Robust Independent ElementaryFeatures (BRIEF), Harris Corners, Edges, Gradient Location andOrientation Histogram (GLOH), Energy of image Gradient (EOG) orTransform Invariant Low-rank Textures (TILT) feature detectionalgorithm. Alternatively, the first party object data and second partyobject data may be obtained based on at least one of an identificationof object data by a user (e.g., the first party), and an instructionfrom another device. The generated or obtained first party object dataand/or second party object data may then be stored (e.g., in objectdatabase 120) at step 708.

At step 710, access to object database 120 is controlled such that thefirst party object data and the second party object data is accessibleto at least the first party and the second party, e.g., via clientdevices 104A and 104B, respectively. The first party and the secondparty may include at least one of a person, a group of people, and anautomated device.

At step 712, modification of the first party object data by the secondparty (e.g., via client device 104B) is facilitated to generate modifiedfirst party object data, the modified first party object data being inaccordance with at least one context parameter of the second partyobject data. For example, modifications of the first party object datacan include the addition of supplemental information, changing one ormore spatial aspects of first party object data, changing an attributeof a first party object data, combining first party object data withsecond party object data, creating a relationship between the firstparty object data and second party object data, changing a relationshipbetween the first party object data and second party object data, andadding contextual data to the first party object data.

At step 714, if the modified first party object data is identified as aknown object, additional object data associated with the known object isretrieved at step 718, e.g., from object database 120 and/or from athird-party source. If the modified first party object data is notidentified as a known object, an image recognition search is performedon the modified first party object data at step 716. For example, theimage recognition search may be performed using an image recognitionalgorithm, wherein the modified first party object data is used as aquery image that is matched to one or more document images of knownobjects. Once a match is found, additional object data associated withthe identified known object may be retrieved at step 718. In someembodiments, as object data is updated over time with more information,the recognition of the real-world object matching the modified firstparty object data can be updated to recognize new or different aspectsof the real-world object, and to further refine the recognizedreal-world object (such as from a general recognition of a car in animage as a generic ‘car’ to increasingly specific recognitions of‘sedan’ to ‘sedan of certain make and model’ to ‘sedan of certain makeand model of a particular year’). As above, the image recognitionalgorithm used for the image recognition search may be a featuredetection algorithm including at least one of a scale-invariant featuretransform (SIFT), Fast Retina Keypoint (FREAK), Histograms of OrientedGradient (HOG), Speeded Up Robust Features (SURF), DAISY, Binary RobustInvariant Scalable Keypoints (BRISK), FAST, Binary Robust IndependentElementary Features (BRIEF), Harris Corners, Edges, Gradient Locationand Orientation Histogram (GLOH), Energy of image Gradient (EOG) orTransform Invariant Low-rank Textures (TILT) feature detectionalgorithm.

At step 720, the modified first party object data and the additionalobject data is communicated to the first party. For example, portalengine 110 can be configured to retrieve the modified first party objectdata (e.g., from local memory and/or object database 120) andcommunicate the modified first party object data (e.g., via network 102using an output module) for presentation to the first party, as shown inFIG. 6. In some embodiments, the modified first party object data alsomay be communicated to the second party. As such, the modified firstparty object data may be usable for presentation within a real-timecollaborative user environment. In various embodiments, the first partycan also modify the second party object data in a similar fashion,thereby generating modified second party object data in the context offirst party object data. For example, portal engine 110 can permit thefirst user to access object database 120 and modify first party objectdata in the context of second party object data in order to generatemodified first party object data. Such modifications to the first partyobject data and/or the second party object data can be made prior to(long-term) storage of the unmodified first party object data and/orsecond party object data. It should be appreciated that in someembodiments modified first party object data and modified second partyobject data can be further modified repeatedly in an iterative process.

In one illustrative example, portal engine 110 may be a 3-D graphicalgame engine. Based on the recognition of a particular scene, includingthe physical attributes of a scene and the real-world objects within ascene, the 3-D graphical game engine can incorporate virtual visualelements in an augmented reality display of a user's device. Therecognized physical attributes of the scene can be used to constructinteraction rules with virtual elements of the augmented realityenvironment. For example, the augmented reality functions can beassociated with gameplay, whereby the virtual elements of the game(e.g., characters, menus, goals, interactions with other game elements,game rules) can be generated based on the scene in which the user deviceis located, and the physical attributes thereof. In further embodiments,two devices in a same general location can both contribute to thegeneration of the game elements based on their individual perceptions ofthe scene and real-world objects within the scene. User devices inseparate locations can also collaborate in game play, whereby users inenvironments providing different scenes with different physicalattributes can generate different virtual game content for the augmentedreality gameplay. For example, if a first player is in a hot-weatherenvironment and a second player is in a cold-weather environment, theaugmented reality content generated by the hot-weather scene can includeheat-based weapons which would be common in a hot-weather environment,but with limited effectiveness against hot-weather enemies. Bycollaborating with the cold-weather player, the cold-weather player canadd cold-weather attributes to the hot-weather player's weapon, makingthe weapon more effective against the hot-weather enemies.

The following is an illustrative example of a use case involvingfunctions and processes associated with the inventive subject matter.Consider a first user, Roy, whose car suddenly starts making strangenoises while he is driving. Roy calls a second user, his friend Frank,who is a car enthusiast with experience working with cars and engines,for help with the problem. Frank wants to help Roy out, and they planfor Frank to gather his tools from his garage, and go to Roy's house.However, before Frank goes through the trouble of gathering all of histools and driving across town, they both want to know whether they'llhave the necessary tools to do the job, as well as the necessary partsfor any necessary repairs.

Roy uses the camera on his cellular phone to take a picture of theengine of his car. Roy can access a portal engine that provides accessto a virtual garage, and provide the image data from his cellular phoneto the portal engine. Using image recognition techniques, the portalengine recognizes the engine in the image data, and obtains object datathat includes information about the make and model of Roy's car, theengine type, and a 3-D graphic mock-up of the engine.

In a variation of this example, Roy can upload a video of the engine ofhis car, including audio of the engine as it is running, and diagnosticdata received from the engine controller. The video and controller datacan be communicated instead of or in addition to the still images. Inanother variation of the example, Roy can record an audio clip of theengine, such as by using his phone, and upload the sound in addition tothe still images and controller data.

Frank, in turn, can take photos of his tools in his own garage, andprovide the image data to the portal engine. The portal enginerecognizes the tools in the images submitted by Frank, and obtainsobject data related to the tools, including information about each ofthe tools (e.g., specification, sizes, gauges, etc.), and a virtualrepresentation of the tools that is integrated into the virtual garage.

Within the virtual garage, Frank can utilize the virtual tools and theobject data obtained from Roy to simulate a repair on the virtualengine. For example, Frank may determine whether the (real-world) toolshe has are the right sizes/gauges for use on the parts of the enginethat need to be worked on, such as whether the wrenches he has will fitthe bolts of the engine, and whether the tools will accommodate thehard-to-reach corners of the engine. Using the virtual tools, Frank mayservice the virtual engine as he would with the real engine to get tothe suspected problem area.

In embodiments where audio (or video with accompanying audio), Frank canuse this audio information to assist in diagnosing potential problemswith the engine. For example, the object data for a particular engine ina database can include representations of the engine, along withcorresponding audio, depicting the engine as affected by a variety ofknown problems typically encountered by the engine (e.g., the sources ofrattling, pings, and other issues that can result in anunusually-sounding engine). In these embodiments, the portal engine canemploy sound recognition techniques to match the sound provided by Royto the proper representation of the engine recognized from the visualimages.

Once the virtual engine diagnostic has been performed, and Frank hasarrived at the suspected cause of the trouble, the portal engine can beused to obtain data objects about the specific parts or enginecomponents that are at fault. The data objects can include suitablereplacements, their cost at various sources, and instructions oninstallation. Frank can further use the virtual tools to install avirtual representation of the replacement part.

In a variation of this example, the portal engine can present anaugmented reality overlay of some or all of the virtual representationof the car's engine over the real-world engine, such as via the displayon Roy's phone. As such, Roy can see the changes Frank makes to thevirtual engine using the virtual tools, overlaid on the real engine. IfRoy has some of the same tools Frank has introduced into the virtualgarage, the augmented reality function can be used as a virtual guidethat walks Roy through the work he can do on the real-world engine onhis own.

The following is another illustrative example of a use case involvingthe inventive subject matter. Consider a scavenger-hunt game played byparticipants that are remotely located from one another.

In this example, a clue in a scavenger hunt can require a combination ofelements by players that are playing together while being physically farapart. For example, a clue in the scavenger hunt can require a piece ofpaper retrieved from a location close to Player A and the use of apencil retrieved from a location close to teammate Player B, who in reallife is across the country from Player A. To be able to retrieve theclue, Player A can use their camera-equipped phone to take a picture ofthe piece of paper at the proper location. Based on the image of thepaper and GPS or other location-identifying data provided by the phone,the portal engine can recognize the paper as the correct one for thescavenger hunt, and obtain object data associated with the use of thepaper in the game. Upon arriving at the appropriate location, Player Bcan use their camera-equipped phone to take a photo of a pencil. Theimage, as well as location information, is similarly provided to theportal engine, which similarly confirms the pencil as a correct pencilfor the game, and obtains object data associated with the pencil. In analternative scenario, the background content of images can be used torecognize the location of the photos of Player A and/or Player B insteadof GPS or other location information.

The object data of the paper and the pencil may include virtualrepresentations of each, which can be communicated for presentation onthe display of each player's phone. The object data can also includeinstructions regarding the clue, such as instructing Player B to write aparticular password on the virtual paper using the virtual pencil (suchas via touch input using the touch-screen of Player B's phone). Afterthe correct phrase is written, the next clue in the scavenger hunt canbe presented to both players.

In another game-related example, Player A may be required to explore adark room. To aid in this part of the game, Player A may be asked totake an image of a candle, e.g., using a mobile device, which isprovided to the portal engine. The portal engine may be configured torecognize the candle, and generate a virtual representation of thecandle (e.g., utilizing an image recognition algorithm) for display onthe devices of Player A and Player B. Player B, may then be asked tofind an object capable of producing fire, such as a set of matches or alighter. Upon finding a lighter, Player B may take an image of thelighter using a mobile device camera and communicate the image to theportal engine. The portal engine may be configured to recognize thelighter and generate a virtual representation of the lighter that PlayerB is able to interact with. Player B can then use the virtualrepresentation of the lighter to light the virtual representation of thecandle. Upon lighting the candle, game functions associated with the litcandle may provide instructions to Player A's mobile device to turn onthe camera's flash or to turn the display of the Player A's device whiteand at maximum brightness, so as to act as a source of light to searchthe dark room.

In a variation of this example, the darkness of the room can be avirtual darkness in addition to a real-world darkness, where the goal ofexploring the dark room is to find an augmented reality item. Thus,while the real-world component of the augmented reality item might befound with any other light source, only the use of the lit virtualcandle would reveal the virtual aspects of the augmented reality item.

Systems, apparatus, and methods described herein may be implementedusing a computer program product tangibly embodied in an informationcarrier, e.g., in a non-transitory machine-readable storage device, forexecution by a programmable processor; and the method steps describedherein, including one or more of the steps of FIGS. 3 and 7 may beimplemented using one or more computer programs that are executable bysuch a processor. A computer program is a set of computer programinstructions that can be used, directly or indirectly, in a computer toperform a certain activity or bring about a certain result. A computerprogram can be written in any form of programming language, includingcompiled or interpreted languages, and it can be deployed in any form,including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment.

A high-level block diagram of an exemplary apparatus that may be used toimplement systems, apparatus and methods described herein is illustratedin FIG. 8. Apparatus 800 comprises a processor 810 operatively coupledto a persistent storage device 820 and a main memory device 830.Processor 810 controls the overall operation of apparatus 800 byexecuting computer program instructions that define such operations. Thecomputer program instructions may be stored in persistent storage device820, or other computer-readable medium, and loaded into main memorydevice 830 when execution of the computer program instructions isdesired. For example, portal engine 110 and object database 120 maycomprise one or more components of apparatus 800. Thus, the method stepsof FIGS. 3 and 7 can be defined by the computer program instructionsstored in main memory device 830 and/or persistent storage device 820and controlled by processor 810 executing the computer programinstructions. For example, the computer program instructions can beimplemented as computer executable code programmed by one skilled in theart to perform an algorithm defined by the method steps of FIGS. 3 and7. Accordingly, by executing the computer program instructions, theprocessor 810 executes an algorithm defined by the method steps of FIGS.3 and 7. Apparatus 800 also includes one or more network interfaces 880for communicating with other devices via a network. Apparatus 800 mayalso include one or more input/output devices 890 that enable userinteraction with apparatus 800 (e.g., display, keyboard, mouse,speakers, buttons, etc.).

Processor 810 may include both general and special purposemicroprocessors, and may be the sole processor or one of multipleprocessors of apparatus 800. Processor 810 may comprise one or morecentral processing units (CPUs), for example. Processor 810, persistentstorage device 820, and/or main memory device 830 may include, besupplemented by, or incorporated in, one or more application-specificintegrated circuits (ASICs) and/or one or more field programmable gatearrays (FPGAs).

Persistent storage device 820 and main memory device 830 each comprise atangible non-transitory computer readable storage medium. Persistentstorage device 820, and main memory device 830, may each includehigh-speed random access memory, such as dynamic random access memory(DRAM), static random access memory (SRAM), double data rate synchronousdynamic random access memory (DDR RAM), or other random access solidstate memory devices, and may include non-volatile memory, such as oneor more magnetic disk storage devices such as internal hard disks andremovable disks, magneto-optical disk storage devices, optical diskstorage devices, flash memory devices, semiconductor memory devices,such as erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), compact disc read-onlymemory (CD-ROM), digital versatile disc read-only memory (DVD-ROM)disks, or other non-volatile solid state storage devices.

Input/output devices 890 may include peripherals, such as a printer,scanner, display screen, etc. For example, input/output devices 890 mayinclude a display device such as a cathode ray tube (CRT), plasma orliquid crystal display (LCD) monitor for displaying information (e.g., aplurality of image transformations for selection) to a user, a keyboard,and a pointing device such as a mouse or a trackball by which the usercan provide input to apparatus 800.

Any or all of the systems and apparatus discussed herein, includingportal engine 110 and object database 120 may be performed by, and/orincorporated in, an apparatus such as apparatus 800.

One skilled in the art will recognize that an implementation of anactual computer or computer system may have other structures and maycontain other components as well, and that FIG. 8 is a high levelrepresentation of some of the components of such a computer forillustrative purposes.

The foregoing specification is to be understood as being in everyrespect illustrative and exemplary, but not restrictive, and the scopeof the invention disclosed herein is not to be determined from thespecification, but rather from the claims as interpreted according tothe full breadth permitted by the patent laws. It is to be understoodthat the embodiments shown and described herein are only illustrative ofthe principles of the present invention and that various modificationsmay be implemented by those skilled in the art without departing fromthe scope and spirit of the invention. Those skilled in the art couldimplement various other feature combinations without departing from thescope and spirit of the invention.

What is claimed is:
 1. A multiparty object recognition system forinteractively manipulating virtual object data, the system comprising: aprocessor; an object database configured to store first party objectdata, captured using a first device associated with a first party, thatcorresponds to at least one structural or property representation of afirst real-world object and further configured to store second partyobject data, captured using a second device associated with a secondparty, that corresponds to at least one structural or propertyrepresentation of a second real-world object; and a memory storingsoftware instructions that, when executed by a processor, cause theprocessor to: obtain the first party object data and the second partyobject data for storage within the object database; control access tothe object database such that the first party object data and the secondparty object data is accessible to at least the first party and thesecond party; generate modified first party object data by facilitatingmodification of the first party object data by the second party, themodified first party object data being in accordance with at least onecontext parameter of the second party object data, wherein the firstparty object data and second party object data are analyzed based onmatching features of the first party object data and the second partyobject data; identify the modified first party object data as arecognized real-world object; retrieve, from the object database,additional object data associated with the recognized real-world object,wherein the additional object data is updatable over time to includeincreasingly specific recognitions of the recognized real-world object;and communicate the modified first party object data for presentation tothe first party, wherein the communication includes at least portions ofthe first party object data and second party object data such that, whendisplayed to the first party, the presentation comprises the at leastone structural or property representation of the first real-world objectand the at least one structural or property representation of the secondreal-world object, wherein the at least one structural or propertyrepresentation of the first real-world object is rendered as beingmanipulated or otherwise modified by the at least one structural orproperty representation of the second real-world object in accordancewith the modified first party object data.
 2. The system of claim 1,wherein the at least one context parameter corresponds to at least oneof a temporal context, a spatial context, a thematic context, anadditive context, a subtractive context, and a geometric context of thesecond party object data.
 3. The system of claim 1, wherein the modifiedfirst party object data is usable for presentation within acollaborative user environment.
 4. The system of claim 1, wherein themodified first party object data is communicated to the first party inreal-time.
 5. The system of claim 1, wherein the modified first partyobject data is communicated as one of machine-readable image data or rawdata which is convertible at a receiving device.
 6. The system of claim1, wherein matching features of the first party object data and thesecond party object data includes using at least one of an imagerecognition algorithm, a pattern recognition algorithm, a speechrecognition algorithm, a user matching algorithm, and a locationrecognition algorithm.
 7. The system of claim 1, wherein at least one ofthe first party object data and the second party object data includes atleast one of digital image data, digital video data, digital audio data,digital text data, known virtual object data, biometric data, financialdata, medical data, and transaction data.
 8. The system of claim 1,wherein at least one of the first party object data and the second partyobject data comprises metadata.
 9. The system of claim 8, wherein themetadata comprises at least one of first party-specific data or secondparty-specific data, location data, time data, user identification data,annotation data, social media data, product data, security data, andcontextual data.
 10. The system of claim 1, wherein the processor isfurther caused to obtain at least one of the first party object data andthe second party object data by using an image recognition algorithm ona digital representation of at least one of the first real-world objectand the second real-world object.
 11. The system of claim 1, wherein theprocessor is further caused to obtain at least one of the first partyobject data and the second party object data based on at least one of anidentification of object data by a user, and an instruction from anotherdevice.
 12. The system of claim 1, wherein the first real-world objectincludes at least one of a visible object, an audible object, abiometric object, a motion, and a gesture.
 13. The system of claim 12,wherein the visible object comprises at least one of a living object, alandmark, an architectural structure, a conveyance, a consumer product,a tool, a printed or displayed object, a toy, a musical instrument, aneducational item, an art object, or a natural inanimate object.
 14. Thesystem of claim 12, wherein the audible object comprises at least one ofa voice, a sound of a musical instrument, a sound of a living object, asound of an inanimate object, a sound of a vehicle, and anelectronically generated audible signal.
 15. The system of claim 1,wherein the first real-world object and second real-world objectcomprise different aspects of a single real-world object.
 16. The systemof claim 1, wherein the first real-world object comprises at least oneproperty that triggers the modification of the first party object data,the at least one property comprising at least one of a chemicalsignature, temperature, pressure, electromagnetic field, radiofrequency,shape, color, texture, taste, smell, mechanical, biometric, orelectrical property.
 17. The system of claim 1, wherein the modificationof the first party object data comprises one or more of addingsupplemental information, changing a spatial aspect of the first partyobject data, changing an attribute of the first party object data,combining the first party object data with the second party object data,creating or changing a relationship between the first party object dataand the second party object data, and adding contextual data to thefirst party object data.
 18. The system of claim 1, wherein themodification of the first party object data includes at least one offiltering, moving, manipulating, combining, editing, annotating, oranimating the first party object data.
 19. The system of claim 1,wherein the modified first party object data comprises at least one ofthe second party object data, and object data from a source other thanthe first party and the second party.
 20. The system of claim 1, whereinthe object database is accessible to at least the first party and thesecond party via a user portal.
 21. The system of claim 20, wherein theuser portal comprises at least one of a mobile device, an appliance, avehicle, a robot, a kiosk, a television, and a game console.
 22. Thesystem of claim 20, wherein the processor is further caused to present auser interface, comprising at least one of a graphical user interface, avoice-controlled interface, and a motion-controlled interface, foraccess to the object database.
 23. The system of claim 1, wherein theprocessor is further caused to control access to the object databasesuch that the object database is accessible to a party other than thefirst party and second party.
 24. The system of claim 23, wherein theprocessor is further caused to control access to the object databasebased on at least one of a classification of a real-world object, aproximity metric, a subscription status, and an authorization.
 25. Thesystem of claim 1, further comprising storing the modified first partyobject data within the object database, wherein the modified firstobject data is further modifiable to generate nth-order modified firstobject data.
 26. The system of claim 1, further comprising communicatingthe modified first party object data to the second party.
 27. The systemof claim 1, wherein the first party and the second party include atleast one of a person, a group of people, and an automated device.
 28. Amultiparty object recognition method for interactively manipulatingvirtual object data, wherein an object database is configured to storefirst party object data, captured using a first device associated with afirst party, that corresponds to at least one structural or propertyrepresentation of a first real-world object and further configured tostore second party object data, captured using a second deviceassociated with a second party, that corresponds to at least onestructural or property representation of a second real-world object, themethod comprising: obtaining the first party object data and the secondparty object data for storage within the object database; controllingaccess to the object database such that the first party object data andthe second party object data is accessible to at least the first partyand the second party; generating modified first party object data byfacilitating modification of the first party object data by the secondparty, the modified first party object data being in accordance with atleast one context parameter of the second party object data, wherein thefirst party object data and second party object data are analyzed basedon matching features of the first party object data and the second partyobject data; identifying the modified first party object data as arecognized real-world object; retrieving, from the object database,additional object data associated with the recognized real-world object,wherein the additional object data is updatable over time to includeincreasingly specific recognitions of the recognized real-world object;and communicating the modified first party object data for presentationto the first party, wherein the communication includes at least portionsof the first party object data and second party object data such that,when displayed to the first party, the presentation comprises the atleast one structural or property representation of the first real-worldobject and the at least one structural or property representation of thesecond real-world object, wherein the at least one structural orproperty representation of the first real-world object is rendered asbeing manipulated or otherwise modified by the at least one structuralor property representation of the second real-world object in accordancewith the modified first party object data.
 29. A computer programproduct embedded in a non-transitory computer readable medium comprisinginstructions executable by a computer processor for interactivelymanipulating virtual object data, wherein an object database isconfigured to store first party object data, captured using a firstdevice associated with a first party, that corresponds to at least onestructural or property representation of a first real-world object andfurther configured to store second party object data, captured using asecond device associated with a second party, that corresponds to atleast one structural or property representation of a second real-worldobject, the instructions being executable by a computer processor toexecute processing comprising: obtaining the first party object data andthe second party object data for storage within the object database;controlling access to the object database such that the first partyobject data and the second party object data is accessible to at leastthe first party and the second party; generating modified first partyobject data by facilitating modification of the first party object databy the second party, the modified first party object data being inaccordance with at least one context parameter of the second partyobject data, wherein the first party object data and second party objectdata are analyzed based on matching features of the first party objectdata and the second party object data; identifying the modified firstparty object data as a recognized real-world object; retrieving, fromthe object database, additional object data associated with therecognized real-world object, wherein the additional object data isupdatable over time to include increasingly specific recognitions of therecognized real-world object; and communicating the modified first partyobject data for presentation to the first party, wherein thecommunication includes at least portions of the first party object dataand second party object data such that, when displayed to the firstparty, the presentation comprises the at least one structural orproperty representation of the first real-world object and the at leastone structural or property representation of the second real-worldobject, wherein the at least one structural or property representationof the first real-world object is rendered as being manipulated orotherwise modified by the at least one structural or propertyrepresentation of the second real-world object in accordance with themodified first party object data.